Engineers use compression (also called coding or encoding) to reduce the bit rate of digital video. Compression decreases the cost of storing and transmitting video by converting the video into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original video from the compressed form. A “codec” is an encoder/decoder system.
A basic goal of compression is to provide good rate-distortion performance. So, for a particular bit rate, an encoder attempts to provide the highest quality of video. Or, for a particular level of quality/fidelity to the original video, an encoder attempts to provide the lowest bit rate encoded video. In practice, considerations such as encoding time, encoding complexity, encoding resources, decoding time, decoding complexity, decoding resources, overall delay, and/or smoothness in quality/bit rate changes also affect decisions made in codec design as well as decisions made during actual encoding.
Perceptible flaws in video after encoding or other processing are sometimes called artifacts in the video, as they result from and show that the encoding or other processing occurred. These artifacts include blocking artifacts, banding artifacts and ringing artifacts.
Block patterns that are introduced by compression and appear in reconstructed video are often called block artifacts. Block artifacts can be especially noticeable in smoothly varying, gradient regions, such as video of a clear sky. Block artifacts result, for example, from splitting a video picture into blocks for encoding, where the encoding includes a frequency transform process and quantization of AC coefficients for the blocks. As bit rate decreases and quantization of AC coefficients becomes more dramatic, block artifacts tend to become worse.
Banding or contouring artifacts occur, for example, when sample values in a picture are converted from a high bit resolution (e.g., 10 bits or 12 bits per sample value) to a lower bit resolution (e.g., 8 bits per sample value). When sample values are clipped to the lower bit resolution, steps between bands of values may become perceptible, especially in regions with smoothly changing sample values (e.g., a gradual transition from light to darker). Suppose a picture includes 10-bit sample values that vary gradually from 65 (dark black) to 74 (lighter black). When converted to 8-bit sample values, the 8-bit values vary between 16 and 18, and noticeable boundaries appear between 16 and 17, and between 17 and 18. Banding artifacts tend to be more noticeable in dark regions and for animation content, computer-generated effects or other content having unnatural, smooth gradients. In practice, banding artifacts are often introduced when high-quality video (e.g., 12-bit or 10-bit 4:4:4/4:2:2 studio-quality 3840×2160 video) is converted to a lower resolution form (e.g., 8-bit 4:2:0 1920×1080 video) for encoding. Many codecs work with 8-bit video.
Ringing artifacts can occur for various reasons. They might occur, for example, when a small object such as a ball moves across a static background of a picture. The ringing artifact appears as a ripple pattern or other band of noise going away from the edge of the artifact into the background of the picture. Such ringing artifacts can result from the frequency transform process and quantization for a block that includes the object or part of the object. Ringing artifacts can also be introduced at edges by excessive sharpening during editing. Other types of ringing artifacts can be introduced during video processing, appearing as repeated vertical or diagonal edges where a dark region changes to light, or vice versa, as a hardware component adjusts to the change in intensity.
Other artifacts such as film scan artifacts and film grain artifacts can result from the process of scanning film into a digital form or from the film medium itself. Film grain artifacts result from grains used to record images in film, and show up as perceptible “graininess” in the video. Some types of video capture devices can also produce artifacts resembling film grain artifacts. Unlike other types of artifacts, which are viewed as undesirable, certain types and amounts of film grain have aesthetic appeal and are deemed desirable by content producers.
Film scan artifacts are introduced during scanning of high-resolution film into a digital format. These artifacts can be caused, for example, by scratches or dust on film or in the film scanning environment, and can show up in the digital video as perceptible smearing or scratches. Or, they can be introduced due to irregularities in the scanning process or scanning equipment, for example, by a faulty or mis-calibrated sensor in a row of sensors, and show up in the digital video as anomalies like “phosphor lines” along a scan direction.
One approach to controlling artifacts in video is to allocate more bit rate to the video during encoding. By using more bits during encoding, artifacts such as block artifacts and some ringing artifacts can be avoided or mitigated. Other types of artifacts can be avoided by careful film scanning and image sharpening before encoding.
Some post-processing approaches to controlling artifacts process video after decoding so as to smooth over or otherwise hide artifacts. Some systems adaptively filter across block boundaries to reduce the visibility of block artifacts. Other systems use dithering during post-processing to adjust the sample values of reconstructed pictures. For example, dithering can introduce small adjustments to values around a jagged edge so that the human viewer “averages” the values and perceives a smoother edge.
While previous approaches to controlling artifacts provide acceptable performance in some scenarios, they do not have the advantages of the techniques and tools described below.